evaluation of recommender systems: a multi-criteria decision making approach

the evaluation and selection of recommender systems is a difficult decision making process. this difficulty is partially due to the large diversity of published evaluation criteria in addition to lack of standardized methods of evaluation. as such, a systematic methodology is needed that explicitly considers multiple, possibly conflicting metrics and assists decision makers to evaluate and find the best recommender system among a given set of alternatives. this paper introduces multi-criteria decision making (mcdm) approach for evaluation of recommender systems. in particular, this paper proposes the use of data envelopment analysis (dea) approach, as a sub-category of mcdm, in order to solve this problem. various dea models are introduced and their applicability are illustrated. a real case of evaluation of recommender systems is used to demonstrate the approach.